Function Test teams are tasked with the validation of a set of features and functions in a system or application. This validation is a comprehensive exercise that demonstrates quality at a feature/functional level in accordance with specifications and requirements placed on the system. Specifically, system testing is used to assess the overall quality of the code from the perspective of the customer.
Testing, such as unit testing, integration testing, system testing, conformance testing, and others is conducted on a complete integrated system to evaluate the entire system's compliance with its specified requirements. This type of testing therefore evaluates the interactions of the various functions in the environment specified in the requirements. System testing is conducted in an environment that attempts to model as realistically as possible scenarios of customer usage. These scenarios are largely based on the quantitative knowledge of domain experts within the test team, as well as continuously gathered information from reported customer experiences with the product.
Function and System Testing is performed on systems that exhibit functional and nonfunctional decisions. For example, consider the table 100 shown in FIG. 1. Table 100 describes examples of supported nonfunctional requirements for Lotus Workplace 3.0 system. The permutations and combinations of systems that can be built around this list are almost endless. Therefore, decisions need to be made prior to testing as to the number of configurations that will be tested (for schedule and resource reasons) whilst at the same time providing as broad a coverage as possible. This determination typically results in some kind of configuration matrix that captures the systems that test teams build and that includes the heterogeneity required to cover a set of nonfunctional attributes.
Typically, configuration plans are built manually and coordinated by a single person. FIG. 2 is an example of a configuration matrix 200 created manually using Lotus. The effort requires multiple passes with many subject matter contributors contributing their own coverage requirements. Once aggregated in a central place, a series of systematic pass-throughs is performed (manually) to change decisions made at various points with a view to ensuring as much heterogeneity and coverage in the decisions made.
This method is time consuming, expensive, tedious, imperfect, and prone to errors. Therefore a need exists to overcome the problems with the prior art as discussed above.